Interacting Dirac Fermions on a Topological Insulator in a Magnetic Field 
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We have studied the fractional quantum Hall states on the surface of a topological insulator thin 
film in an external magnetic field, where the Dirac fermion nature of the charge carriers have been 
experimentally established only recently. Our studies indicate that the fractional quantum Hall 
states should indeed be observable in the surface Landau levels of a topological insulator. The 
strength of the effect will however be different, compared to that in graphene, due to the finite 
thickness of the topological insulator film and due to the admixture of Landau levels of the two 
surfaces of the film. At a small film thickness, that mixture results in a strongly non-monotonic 
dependence of the excitation gap on the film thickness. At a large enough thickness of the film, the 
excitation gap in the lowest two Landau levels are comparable in strength. 



In recent years, there has been an upsurge of interest 
on the unusual magnetic properties of Dirac fermions in 
condensed matter systems. Graphene @, |1] is one such 
system where those properties were exhaustively inves- 
tigated theoretically [1] and experimentally [U, Q . The 
most remarkable of those are the existence of a Landau 
level at zero energy having no magnetic field dependence, 
and the square root dependence of higher energies on the 
Landau level (LL) index and the magnetic field 0, @. 
A direct consequence of these is the unusual half-integer 
quantum Hall effect (QHE). The role of Coulomb intera- 
cions between Dirac fermions in graphene, in the frac- 
tional QHE (FQHE) regime, was first studied by us the- 
oretically j] . The effect was later observed in the exper- 
iments [H |9| . 

Linear dispersion relation in the band structure of 
massless Dirac fermions, first observed in monolayer 
graphene 0, d, [Io| , is also expected to be present in spe- 
cial insulators with topologically protected surface states 
[Til [l2j . The surface states in these topological insulators 
(TIs) are gapless with linear relativistic dispersion rela- 
tion, and therefore bear a close analogy, albeit with some 
major differences [l3j], to the energy spectra in graphene. 
In an external magnetic field, the energetics of the sur- 
face Landau levels in the TIs is also similar to that of the 
Landau levels in graphene. Recently, a direct observa- 
tion of Landau quantization in a TI, Bi 1 _ a .Sb a , thin film, 
was reported, which was found to have the same behavior 
as in graphene, but are found to be suppressed by sur- 
face impurities, thereby indicating its two-dimensional 
(2D) nature [UGl]. Althou g n the low 

-energy dynam- 
ics of the surface states is similar to that of graphene, 
there is an important difference between these two sys- 
tems. While the electronic states in graphene are strictly 
two-dimensional and localized within a single graphene 
layer, the surface states of a TI have finite width in the 
growth direction. This modifies the inter-electron inter- 
action potential and changes the properties of the FQHE 
states. In traditional non-relativistic electron systems, an 
increase in width of the 2D layer results in a reduction 



of the FQHE gaps, which in turn reduces the stability 
of the corresponding incompressible states [lj|. There- 
fore, we should expect that the FQHE gaps in the TIs 
are also influenced by the finite width, in contrast to the 
case in graphene. Experimental observation of the Lan- 
dau quantized topological surface states, in particular the 
field- independent LL at the Dirac point Q [l5| , the sig- 
nature effect of Dirac fermions, has paved the way for 
future observation of the QHE in the surface states of 
the TI. The nature of the FQH states in such a system 
is the subject of this Letter. 

To analyze the properties of the FQHE in the surface 
states of the TIs, we begin with the low-energy effective 
Hamiltonian already introduced in the literature [171 . |18| . 
The Hamiltonian has the matrix form of size 4x4 and 
is given by 
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where a i are the Pauli matrices, e(p) = C x + D lPz + 

DM+Pll M(P) = Mo - B lP 2 z - B 2 (pl+Pl)- Here 
we consider a prototypical TI, Bi 2 Se 3 , with one surface 
state containing a single isotropic Dirac cone at the cen- 
ter of the Brillouin zone. For this system the constants 
are [ij], A x = 2.2 eV-A, A 2 = 4.1 eVA, B 1 = 10 eVA 2 , 
B 2 = 56.6 eVA 2 , C x = -0.0068 eV, D x = 1.3 eVA 2 , 
D 2 = 19.6 eV-A 2 , and M = 0.28 eV. The topological 
insulator film has a finite thickness of L z , i.e., the two 
surfaces of the film are at z = and z — L z . The four- 
component wavefunctions corresponding to the Hamilto- 
nian ([T]) determine the amplitudes of the wavefunctions 
at the positions of Bi and Se atoms: (BLj., Se^, Bi^, Se^), 
where the arrows indicate the direction of the electron 
spin. 

The external magnetic field, B, along the z-direction 
produces surface Landau levels which can be found from 
the Hamiltonian matrix by replacing the x and y compo- 
nents of the momentum by the generalized momentum 
[201 ] and introducing the Zeeman energy, A z — \g s \x B B. 
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FIG. 1: (a) The lowest surface Landau levels of a TI film 
shown for the film thickness L z = 30 A. For each n there are 
two LLs of the TI film, belonging to two surfaces of the film, 
(b) The electron density along the z axis for one of the n = 1 
Landau levels and for different values of the thickness, L z , of 
the Bi 2 Se 3 film. The numbers next to the lines are the values 
of L z . The magnetic field is at 15 Tesla. 



Hereos ~ 8 is the effective g-factor of surface states 
[ttI Il9l | and \i B is the Bohr magneton. For the surface 
Landau level n the wavefunctions are of the form 
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for n > 0. For n — 0, the functions xi=o anc ^ Xn=o vanish 
identically. Here <p n m are wavefunctions of conventional 
non-relativistic LLs with index n and some other intra- 
Landau level index, m, for example, the z component of 
the angular momentum. The functions Xn (z) satisfy the 
following eigenvalue equation 



£Xn{z) 




\/2(n+l) . 
i -n-n 



C x + D 2 (2n + \)ltl - ± A z and 



where e^n 

M Zin = M - B 2 (2n + l)J il -B^. Here £ Q = ^KfcB 

is the magnetic length and x n = (Xn ,Xn\xn >Xn ) T - 
The solution of the system of equations (J3|) determines 



the Landau energy spectrum of the system and the 
width in the z-direction of the Landau level wavefunc- 
tions. This width depends on the Landau level index, 
the strength of the magnetic field, and the thickness of 
the TI film. Each surface of the TI film has a corre- 
sponding set of surface LLs. The LLs of two surfaces 
of TI arc coupled, which is more pronounced for a nar- 
row TI film, L z < 30 A. For TI films of large thick- 
ness, the LLs of two surfaces can be considered as decou- 
pled. In Fig. [1] we illustrate the properties of the surface 
LLs, which were obtained from the numerical solution of 
Eq. ([3]). In what follows, we consider only the lowest sur- 
face LLs with indices n = and n = 1. Strong FQHEs 
can be observed only in these LLs. In Fig. HJa), the LL 
energy spectrum is shown as a function of the magnetic 
field, B, for a TI film of thickness L z = 30 A. The VB- 
dependence of the energy of n = 1 (n = —1) LLs is 
visible, which is a specific property of the relativistic dis- 
persion law of the surface states of the TI. For each value 
of n there are two degenerate surface LLs, belonging to 
the two surfaces. The degeneracy of these LLs is lifted 
due to their coupling. For n = 0, the inter-Landau level 
coupling is relatively weak and the n = LL remains 
almost doubly degenerate. The coupling of the surface 
LLs is illustrate in Fig. [ljb), where the electron density, 

p(z) = |xl 1} | 2 + |xl 2) | 2 + \x { n 3) \ 2 + lxi 4) | 2 , is shown for dif- 
ferent thicknesses of the TI film and for one of the n = 1 
LLs. For a small thickness of the TI film the surface LLs 
strongly overlap in space, which results in a strong inter- 
LL coupling and large electron density within the whole 
TI film. With increasing film thickness the surface LLs 
become decoupled with strong localization at the surfaces 
of the TI film and zero electron density in the bulk region 
of the film. 

In the FQHE regime the electrons partially occupy a 
single LL and the properties of such a system are com- 
pletely determined by the electron-electron interaction 
potential within the corresponding LL [l6j]. The interac- 
tion potential projected on a given LL is determined by 
the Haldane pseudopotentials [2l|, which are the ener- 
gies of two electrons with relative angular momentum m. 
With the known wavefunctions © of the LL with index 
n the pseudopotentials can be readily evaluated. 

Just as in graphene, the n = Landau level in a TI 
is identical to the n = traditional (non-relativistic) 
Landau level but with a finite width, while the n > 
LL in the TI is a mixture of the n and n — 1 non- 
relativistic states. In graphene, such a mixture results 
in the strongest FQHE being in the n — 1 LL 0]. Simi- 
lar behaviour is also expected in a TI film. To study the 
properties of the FQHE in a TI we numerically evalu- 
ate the energy spectrum of a finite JV-electron system in 
the spherical geometry [2l|. The radius of the sphere is 
^/SIq, where 25* is the number of magnetic fluxes through 
the sphere in units of the flux quantum. For a given num- 
ber of electrons, the radius of the sphere determines the 
filling factor of the system. For example, the v = 1/m 
FQHE (m is an odd integer) corresponds to the relation 
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FIG. 2: (a) The v = \ FQHE gap, shown for different Lan- 
dau levels of a TI film as a function of the film thickness. 
The magnetic field is 15 Tesla. The excitation gaps were ob- 
tained numerically for a finite-size system of N = 9 electrons 
and the parameter of the sphere S = 12 [2l| . The energy is 
shown in units of the Coulomb energy, e c — e 2 /k£ q . The red 
arrow indicates the FQHE gap in the n = Landau level of 
graphene. (b) The electron densities P\~ (z) and p^^^z) 
of two n — 1 Landau levels at a film thickness of L z = 24 A. 
The black (solid and dotted) lines correspond to the n = 1 
Landau level in panel (a) (black line), while the blue (solid and 
dotted) lines correspond to the n = 1 Landau level in panel 
(a) (blue line). The densities p{ n=1) (z) = ) (z)| 2 + |Xn ) {z)\ 2 
and p[ n ~°\z) — \%n i z )\ 2 + \Xn (z)\ 2 show the occupations of 
the n = 1 and n = non-relativistic Landau level functions, 
respectively. 



S = (y)(7V— 1). The corresponding collective excitation 
gap determines the stability of the FQHE (l6| . 

The specific properties of the surface LLs in TI films 
that modify the inter-electron interaction strength within 
a single LL and change the stability of corresponding 
FQHE are, (i) the intcr-LL coupling of LLs of two sur- 
faces modifies the structure of the corresponding wave- 
functions and changes the mixture of the n and n — 1 
non-relativistic LL functions. This effect is visible only 
for the n — 1 TI Landau level and is the strongest at small 
thickness of the TI film, for which the inter-LL mixture is 
strong (see Fig. [lj , and (ii) the finite thickness of the TI 
film results in a finite width of the wavefunctions, which 
reduces the interaction strength and hence reduces the 
FQHE gaps. 




Magnetic field (Tesla) 



FIG. 3: The v = \ FQHE gap for different Landau levels of 
the TI film: two n — 1 LLs (black lines) and one n = LL 
(red line). The thickness of the film is (a) L z = 25 A and 
(b) L z — 50 A. The red arrows show the FQHE gaps in the 
n — and n = 1 Landau levels of graphene. The FQHE gaps 
were obtained numerically for a finite-size system of N = 9 
electrons and the parameter of sphere S — 12. The energy is 
shown in units of the Coulomb energy, e c = e 2 /k£ . 



To illustrate the manifestation of these two effects, we 
present in Fig. [5] the dependence of the v = | excitation 
gap on the thickness of the TI film for n = and n = 1 
LLs and a fixed magnetic field. For large thickness of 
the TI film (see Fig. EJa)) the inter-LL coupling is weak 
and the surface LLs are decoupled. In this case the main 
effect of the TI film on the FQHE states is through the 
finite width of the LL wavefunctions, which results in a 
reduction of the FQHE gaps. The monotonic reduction 
of the gaps for both n = and n — 1 TI Landau levels is 
clearly visible in FigJ5Ja) for large L z . The corresponding 
gap in a thicker TI film is usually less than the gap in n = 
LL of graphene (shown by the red arrow in Fig. [5Ja)). 
The reduction is more pronounced for the n = 1 LL and 
for large L z the gaps in the n — 1 and n — LLs become 
comparable. Although the reduction of FQHE in a thick 
TI film is relatively large, the FQHE should be observable 
with a larger gap at the n = 1 surface LLs. 

At a small thickness of the TI film, the inter-LL mix- 
ture becomes strong, which modifies considerably the 
behavior of the FQHE gaps in the n = 1 LLs. Since 
the n = TI Landau level consists of the n = non- 
relativistic LL wavefunction, the mixture of surface LLs 
has an weak effect on the interaction strength and the 
FQHE gaps in the n = LLs. As a result, the FQHE 
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gap in the n = LL monotonically decreases with the TI 
film thickness within the whole range of L z due to the 
finite width of the LL wavefunctions. 

The n = 1 TI Landau level shows a different behavior. 
Since the n = 1 TI Landau level is a mixture of the n = 
and n = 1 non-relativistic LL wavefunctions, the interac- 
tion potential becomes sensitive to the inter-LL coupling 
because such a coupling modifies the contribution of the 
ri = and n = 1 non-relativistic LL wavefunctions. As 
a result two n = 1 TI Landau levels, although having al- 
most the same energy, show different behavior as a func- 
tion of L z . The difference between the two n = 1 LLs 
is illustrated in Fig.^b), which shows the contributions 
of the n — and n = 1 non-relativistic LL functions to 
the TI LLs. Clearly, one of the n = 1 TI LL (black lines) 
has a large contribution from the n = non-relativistic 
LL functions, while the other n = 1 TI LL (blue line) 
has a large contribution from the n = 1 non-relativistic 
LL functions. Due to these different contributions the 
FQHE gap of one of the n = 1 LLs is large (almost equal 
to the gap of n = 1 LL in graphene) for small L z and 
decreases with increasing thickness of the TI film. The 
FQHE gap at the other n = 1 LL is small at small L z and 
with increasing L z shows a well-pronounced maximum at 
L z ~ 30 A (see Fig. E^a)). 

Another way to see these different properties of small 
and large thickness of the TI film is to study the depen- 
dence of the FQHE gaps on the magnetic field. In Fig. [3] 
we present the dependence of the v = \ FQHE gap on the 
magnetic field for two TI films with different thicknesses, 
L z = 25 A (small), and L z — 50 A (large). For a large 
thickness of the TI film, L z = 50 A(see Fig. G^b)) the 
surface LLs are decoupled. In this case the finite width 
of the LL wavefunctions results in a monotonic reduction 
of the FQHE gaps for both n = and n — 1 LLs. The 
strength of such a reduction is characterized by the di- 
mensionless width of the LL wavefunctions, expressed in 



units of the magnetic length, £ . With increasing mag- 
netic field, the magnetic length decreases resulting in an 
increase of the dimensionless width of the LL wavefunc- 
tions. 

At a small thickness of the TI film, the inter-LL mix- 
ture is strong, which does not affect the FQHE gap in the 
n = LL but modifies the behavior of the FQHE gaps 
in the n = 1 LL. For one of the n = 1 LLs the FQHE 
gap decreases with increasing magnetic field, while for 
the other n = 1 LL the FQHE gap increases with the 
magnetic field. Such an increase is the direct manifesta- 
tion of inter-LL coupling. For both n = 1 LLs the FQHE 
gap lies between the FQHE gaps of n = and n = 1 LLs 
of graphene. 

Therefore, the FQHE can indeed be observed in the 
surface Landau levels of a topological insulator. The 
strength of the FQHE, which is characterized by the value 
of the excitation gap, is however reduced somewhat with 
increasing thickness of the TI film. This is due to an 
increase of the width in the ^-direction of the surface 
Landau level wavefunctions. The finite thickness of the 
TIs notwithstanding, the FQHE gaps are the largest in 
the n = 1 Landau level, which is similar to the case of a 
monolayer graphene Q ■ The reduction of the FQHE gaps 
is more pronounced in the n = 1 Landau level. For the 
n = 1 TI Landau levels the dependence of the FQHE gaps 
on the thickness of the film is strongly non-monotonic at 
a small thickness, which is due to strong inter-LL cou- 
pling. At a large enough thickness of the film, L z > 7 
nm, the gaps of the FQHE states in the n = and n = 1 
Landau levels become comparable. Experimental obser- 
vation of these theoretical predictions, just as in the case 
of graphene 0-Q j would be an important advancement 
in understanding the nature of correlated Dirac fermions 
in this unique state of matter. 
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